Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Parse rustls.h to generate an API docs website #510

Open
wants to merge 6 commits into
base: main
Choose a base branch
from

Conversation

cpu
Copy link
Member

@cpu cpu commented Dec 19, 2024

librustls: add missing API docs

A few elements of our public API had no associated docs. This is going to be flagged as an error by the API doc generation tooling, so let's fix it up-front with some short doc additions.

add docgen tools binary

This commit adds a docgen binary to the tools crate. This tool can generate a .json file describing the API based on parsing the cbindgen generated rustls.h header file using a tree-sitter C grammar.

The produced JSON can in turn be used to generate markdown web documentation, or any other format required.

Along with generating docs this tool applies some basic policy. In particular it requires that all public API items be documented or it will produce an error instead of the JSON data. We can extend this in the future to require (for example) describing arguments and return values in doc comments.

The generated JSON has the following structure:

{
  "structs": [ ... ],
  "functions": [ ... ],
  "callbacks": [ ... ],
  "enums": [ ... ],
  "externs": [ ... ],
  "aliases": [ ... ]
}

Where each key is a general category of item:

  • Structure definitions
  • API function prototypes
  • Callback typedefs
  • Enum typedefs
  • Extern typedefs
  • Simple type alias typdefs

Within each category items are described like:

{
  "anchor": "rustls-accepted",
  "comment": "A parsed ClientHello produced by a ...",
  "name": "rustls_accepted",
  "text": "```c\nstruct rustls_accepted\n```"
},

The anchor field is useful for creating anchor links that identify the item. The name is the actual name of the item. The comment is the C block comment content (with the block comment syntax removed, but other whitespace left as-is). Lastly the text field holds the actual text that defines the item in rustls.h (without comment) as a C formatted markdown code block.

add an API docs website powered by Zola

This commit adds a Zola website that can template the API docs JSON
data into a nice website showing off our API.

The choice of Zola (and the initial CSS/config) are inspired by the https://rustls.dev/ website which also uses Zola. In the future we can add other pages (e.g. higher level tutorial/getting started content).

To use:

cargo run --bin docgen > website/static/api.json
cd website && zola serve
open http://locahost:1111

ci: add docs page deployment workflow

Lifted from the equivalent config in the rustls repo, and customized for rustls-ffi.

Updates #215

@cpu cpu self-assigned this Dec 19, 2024
@cpu
Copy link
Member Author

cpu commented Dec 19, 2024

ci: add docs page deployment workflow

Not sure the best way to demonstrate this working without merging first. Probably running zola locally is the best bet for now?

@cpu
Copy link
Member Author

cpu commented Dec 19, 2024

rustls-ffi / Build+Test (ubuntu-latest, clang, nightly, aws-lc-rs) (push) Failing after 3m

Odd. Out of time for today but I'll look into this soon.

@cpu cpu marked this pull request as draft December 19, 2024 23:00
@cpu cpu force-pushed the cpu-api-docs-clean branch 2 times, most recently from bde927d to f0c393e Compare December 20, 2024 19:14
@cpu
Copy link
Member Author

cpu commented Dec 20, 2024

rustls-ffi / Build+Test (ubuntu-latest, clang, nightly, aws-lc-rs) (push) Failing after 3m

Odd. Out of time for today but I'll look into this soon.

Not so odd. I broke the ech_fetch util invocation moving things around, and also needed to exclude ECH tests from the MSRV runs because the tools workspace w/ ech_fetch needs a higher MSRV for tree-sitter.

@cpu
Copy link
Member Author

cpu commented Dec 20, 2024

Here's a preview I made with a test repo: https://cpu.github.io/rustls-ffi-docs/

There's definitely room for improvement. In particular there's no cross-linking yet, and I think a comprehensive edit pass through the existing comments in the .h would be helpful.

@cpu cpu force-pushed the cpu-api-docs-clean branch from f0c393e to a2b6c53 Compare December 20, 2024 20:08
Copy link
Collaborator

@jsha jsha left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Haven't reviewed the code in detail but the concept and the output look good to me 👍🏻

Base automatically changed from cpu-workspace to main December 21, 2024 14:55
@cpu
Copy link
Member Author

cpu commented Dec 21, 2024

Haven't reviewed the code in detail but the concept and the output look good to me

Thanks for taking a look, much appreciated 🙇

@cpu cpu force-pushed the cpu-api-docs-clean branch from a2b6c53 to 1c2a3ec Compare December 21, 2024 16:59
@cpu cpu marked this pull request as ready for review December 21, 2024 16:59
Copy link
Member

@ctz ctz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sorry, these review comments were trapped in "finish your review" purgatory

.github/workflows/test.yaml Show resolved Hide resolved
tools/src/bin/docgen/main.rs Show resolved Hide resolved
website/config.toml Outdated Show resolved Hide resolved
cpu added 4 commits December 23, 2024 14:02
On UNIX, use `tee` to get the output to stdout and `grep` the stream
afterwards without using an output file.

For Windows, use powershell's `tee` equiv (`Tee-Object`) and display the
log file before grepping it.

This will surface any errors that occur alongside the failed grep exit code.
A few elements of our public API had no associated docs. This is going
to be flagged as an error by the API doc generation tooling, so let's
fix it up-front with some short doc additions.
In a subsequent commit we're adding a docgen tool to the tools crate
with a dep that requires a higher MSRV than the main crate.

To make this less painful this commit excludes the tools crate from the
workspace default list and skips usage of the ech-fetch tool for the
MSRV runs.
This commit adds a `docgen` binary to the tools crate. This tool can
generate a .json file describing the API based on parsing the `cbindgen`
generated `rustls.h` header file using a tree-sitter C grammar.

The produced JSON can in turn be used to generate markdown web
documentation, or any other format required.

Along with generating docs this tool applies some basic policy. In
particular it requires that **all** public API items be documented or it
will produce an error instead of the JSON data. We can extend this in
the future to require (for example) describing arguments and return
values in doc comments.

The generated JSON has the following structure:

```json
{
  "structs": [ .. ],
  "functions": : [ .. ],
  "callbacks": [ .. ],
  "enums": [ .. ],
  "externs": [ .. ],
  "aliases": [ .. ]
}
```

Where each key is a general category of item:

* Structure definitions
* API function prototypes
* Callback typedefs
* Enum typedefs
* Extern typedefs
* Simple type alias typdefs

Within each category items are described like:

```json
{
  "anchor": "rustls-accepted",
  "comment": "A parsed ClientHello produced by a ...",
  "name": "rustls_accepted",
  "text": "```c\nstruct rustls_accepted\n```"
},
```

The anchor field is useful for creating anchor links that identify the
item. The name is the actual name of the item. The comment is the
C block comment content (with the block comment syntax removed, but
other whitespace left as-is). Lastly the text field holds the actual
text that defines the item in rustls.h (without comment) as
a C formatted markdown code block.
@cpu cpu force-pushed the cpu-api-docs-clean branch from 1c2a3ec to d0eaa12 Compare December 23, 2024 19:22
@cpu
Copy link
Member Author

cpu commented Dec 23, 2024

rustls-ffi / Windows (aws-lc-rs, Debug) (push) Failing after 5m

Filed #514 for this. There's something flaky going on for this test on Windows runners.

cpu added 2 commits December 26, 2024 15:19
This commit adds a Zola[0] website that can template the API docs JSON
data into a nice website showing off our API.

The choice of Zola (and the initial CSS/config) are inspired by the
https://rustls.dev website which also uses Zola.

To use:
```
cargo run --bin docgen > website/static/api.json
cd website && zola serve
open http://locahost:1111
```

[0]: https://www.getzola.org/
Lifted from the equivalent config in the rustls repo, and customized for
rustls-ffi.
@cpu cpu force-pushed the cpu-api-docs-clean branch from d0eaa12 to c183852 Compare December 26, 2024 20:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants